System for gaze interaction

ABSTRACT

The present invention provides improved methods and systems for assisting a user when interacting with a graphical user interface by combining gaze based input with gesture based user commands. The present invention provide systems, devices and method that enable a user of a computer system without a traditional touch-screen to interact with graphical user interfaces in a touch-screen like manner using a combination of gaze based input and gesture based user commands. Furthermore, the present invention offers a solution for touch-screen like interaction using gaze input and gesture based input as a complement or an alternative to touch-screen interactions with a computer device having a touch-screen, such as for instance in situations where interaction with the regular touch-screen is cumbersome or ergonomically challenging. Further, the present invention provides systems, devices and methods for combined gaze and gesture based interaction with graphical user interfaces to achieve a touchscreen like environment in computer systems without a traditional touchscreen or in computer systems having a touchscreen arranged ergonomically unfavourable for the user or a touchscreen arranged such that it is more comfortable for the user to use gesture and gaze for the interaction than the touchscreen.

FIELD OF THE INVENTION

The invention generally relates to computer implemented systems andmethods for utilizing detection of eye movements in connection withinteractive graphical user interfaces. In particular, the presentinvention relates to systems and methods for assisting a user wheninteracting with a graphical user interface by combining eye based inputwith gesture based input and gesture based user commands.

BACKGROUND

Human computer interaction has been revolutionized by the introductionof the graphical user interface (GUI). Thereby, an efficient means wasprovided for presenting information to a user with a bandwidth thatimmensely exceeded any prior channels. Over the years the speed at whichinformation can be presented has increased further through colourscreens, enlarged displays, intelligent graphical objects (e.g. pop-upwindows), window tabs, menus, toolbars, etc. During this time, however,the input devices have remained essentially unchanged, i.e. the keyboardand the pointing device (e.g. the mouse, track ball or touchpad). Inrecent years, handwriting devices have been introduced (e.g. in the formof a stylus or graphical pen). Nevertheless, while output bandwidth hasmultiplied several times, the input bandwidth has been substantiallyunchanged. Consequently, a severe asymmetry in the communicationbandwidth in the human computer interaction has developed.

In order to decrease this bandwidth asymmetry as well as to improve andfacilitate the user interaction, various attempts have been made to useeye-tracking for such purposes. By implementing an eye tracking devicein e.g. a laptop, the interaction possibilities between the user and thedifferent software applications run on the computer can be significantlyenhanced.

Hence, one interesting idea for improving and facilitating the userinteraction and for removing the bandwidth asymmetry is to use eye gazetracking instead or as a complement to mouse input. Normally, the cursoris positioned on the display according to the calculated point of gazeof the user. A number of different techniques have been developed toselect and activate a target object in these systems. In one example,the system activates an object upon detection that the user fixates hisor her gaze at a certain object for a certain period of time. Anotherapproach is to detect an activation of an object when the user's eyeblinks.

However, there are problems associated with these solutions using eyetracking. For example, the humans use their eye in perceptive actionsinstead of controlling. Therefore, it may be stressful to carefully useeye movements to interact with a computer, for example, to activate andselect an object presented on the display of the computer. It may alsobe difficult to control blinking or staring in order to interact withobjects presented on a display.

Thus, there is a need within the art for improved techniques that enableuser interaction with a computer provided with an eye tracking deviceallowing the user to control, select and activate objects and parts ofobjects presented on a display of the computer using his or her eyes ina more intuitive and natural way. Furthermore, there is also a needwithin the art for techniques that in a more efficient way takesadvantage the potential of using eye tracking for improving andfacilitating the user interaction with a computer.

One such attempt is presented in US pat. appl. (publication number2005/0243054) to Beymer et al. in which a technology for selecting andactivating a target object using a combination of eye gaze and keypresses is disclosed. More specifically, a user looks at a targetobject, for example, a button on a graphical user interface and thenpresses a selection key of the keyboard. Once the selection key ispressed, a most probable target is determined using probabilityreasoning. The determined target object is then highlighted and the usercan select it by pressing the selection key again. If the highlightedobject is not the target object, the user can select another targetobject using additional keys to navigate to the intended target object.

However, this technology is limited to object selection and activationbased on a combination of eye gaze and two sequential presses of onededicated selection key.

In U.S. Pat. No. 6,204,828 to Amir et al., a computer-driven system foraiding a user to positioning a cursor by integrating eye gaze and manualoperator input is disclosed. A gaze tracking apparatus monitors the eyeorientation of the user while the user views a screen. Concurrently, thecomputer monitors an input device, such as a mouse, for mechanicalactivation by the operator. When the computer detects mechanicalactivation of the input device, it determined an initial cursor displayposition within a current gaze area. The cursor is then displayed on thescreen at the initial display position and thereafter the cursor ispositioned manually according to the user's handling of the input devicewithout regard to the gaze.

Consequently, there still remains a need within the art of an improvedtechnique that in a more efficient way takes advantage of the potentialin using eye tracking for improving and facilitating the userinteraction with a computer and in particular user interaction withgraphical user interfaces.

SUMMARY

An object of the present invention is to provide improved methods,devices and systems for assisting a user when interacting with agraphical user interface by combining gaze based input with gesturebased user commands.

Another object of the present invention is to provide methods, devicesand systems for user friendly and intuitive interaction with graphicaluser interfaces.

A particular object of the present invention is to provide systems,devices and methods that enable a user of a computer system without atraditional touch-screen to interact with graphical user interfaces in atouch-screen like manner using a combination of gaze based input andgesture based user commands. Furthermore, the present invention offers asolution for touch-screen like interaction using gaze input and gesturebased input as a complement or an alternative to touch-screeninteractions with a computer device having a touch-screen, such as forinstance in situations where interaction with the regular touch-screenis cumbersome or ergonomically challenging.

Another particular object of the present invention is to providesystems, devices and methods for combined gaze and gesture basedinteraction with graphical user interfaces to achieve a touchscreen likeenvironment in computer systems without a traditional touchscreen or incomputer systems having a touchscreen arranged ergonomicallyunfavourable for the user or a touchscreen arranged such that it is morecomfortable for the user to use gesture and gaze for the interactionthan the touchscreen.

In the context of the present invention, the term “GUI” (Graphical User

Interface) refers to a graphics-based user interface with pictures orimages and words (including e.g. signs and figures) on a display thatincorporate, for example, movable windows and icons.

Further, in the context of the present invention the terms “object” or“object part” refer to an interactive graphical object or GUI objectsuch as a window, an icon, a button, a scroll bar, a hyperlink, ornon-interactive objects such as an image, text or a word in a text thatthe user desires to select or activate.

In the context of the present invention, the term “touch pad” (or theterm “trackpad”) refers to a surface sensor for detecting the positionand movement of one or multiple fingers and/or one or multiple otherobjects intended for pointing, drawing or making gestures, such as forinstance a stylus.

These and other objects of the present invention are achieved by meansof a system having the features defined in the independent claims.Embodiments of the invention are characterized by the dependent claims.

According to an aspect of the present invention, there is provided acontrol module for implementation in, for example, a computer device orhandheld device or a wireless transmit/receive unit (WTRU) for handlingand generating gesture based control commands to execute user actionbased on these commands. The control module is configured to acquireuser input from input means adapted to detect user generated gesturesand gaze data signals from a gaze tracking module and to determine atleast one user generated gesture based control command based on the userinput. Further, the control module is configured to determine a gazepoint area on the information presentation area including the user'sgaze point based on at least the gaze data signals and to execute atleast one user action manipulating a view presented on the graphicalinformation presentation area based on the determined gaze point areaand at least one user generated gesture based control command, whereinthe user action is executed with the determined gaze point area as astarting point. The gaze point area serving as a starting point may bean area at which the user initially gazes at or a fine tuned area, i.e.an area that the user has selected by tuning or correcting commands via,for example, the input means, thereby correcting or tuning an initialgaze point area to a selected area.

According to another aspect of the present invention, there is provideda method for generating gesture based commands during user interactionwith an information presentation area, for example, associated with orincluded in a computer device or handheld device, or associated with orincluded in a wireless transmit/receive unit (WTRU). The methodcomprises acquiring user input corresponding to user generated gesturesand gaze data signals and determining at least one user generatedgesture based control command based on the user input. Further, a gazepoint area on the information presentation area including the user'sgaze point is determined based on at least the gaze data signals and atleast one user action manipulating a view presented on the informationpresentation area is executed based on the determined gaze point areaand at least one user generated gesture based control command, whereinthe user action is executed with the determined gaze point area as astarting point.

According to a further aspect of the present invention, there isprovided a handheld portable device provided with or associated with aninformation presentation area and comprising input means adapted todetect user generated gestures and a gaze tracking module adapted todetect gaze data of a viewer of the information presentation area. Thehandheld device further comprises a control module configured to acquireuser input from the input means and gaze data signals from the gazetracking module and to determine at least one user generated gesturebased control command based on the user input. The control module isfurther configured to determine a gaze point area on the informationpresentation area including the user's gaze point based on at least thegaze data signals and to execute at least one user action manipulating aview presented on the information presentation area based on thedetermined gaze point area and at least one user generated gesture basedcontrol command, wherein the user action is executed with the determinedgaze point area as a starting point. In embodiments of the presentinvention, the handheld device may be a cellular phone, a smartphone, aniPad or similar device, a tablet, a phoblet/phablet, a laptop or similardevice.

According to a further aspect of the present invention, there isprovided a wireless transmit/receive unit, WTRU, associated with aninformation presentation area and comprising input means adapted todetect user generated gestures and a gaze tracking module adapted todetect gaze data of a viewer of the information presentation area. TheWTRU further comprises a control module configured to acquire user inputfrom the input means and gaze data signals from the gaze tracking moduleand to determine at least one user generated gesture based controlcommand based on the user input. The control module is furtherconfigured to determine a gaze point area on the informationpresentation area including the user's gaze point based on at least thegaze data signals and to execute at least one user action manipulating aview presented on the information presentation area based on thedetermined gaze point area and at least one user generated gesture basedcontrol command, wherein the user action is executed with the determinedgaze point area as a starting point.

The term “wireless transmit/receive unit (WTRU)” include but is notlimited to a user equipment (UE), a mobile station, a fixed or mobilesubscriber unit, a cellular telephone, a smartphone, a personal digitalassistant (PDA), a computer, or any other type of device capable ofoperating in a wireless environment such as a wireless local areanetwork (WLAN) or wireless mobile communication system (e.g. a thirdgeneration (3G) global system for mobile communication and systems formobile communication including long term evolution (LTE) cells).

According to another aspect of the present invention, there is provideda system for user interaction with an information presentation area. Thesystem comprises input means adapted to detect user generated gesturesand a gaze tracking module adapted to detect gaze data of a viewer ofthe information presentation area. Further, the system includes acontrol module configured to acquire user input from the input means andgaze data signals from the gaze tracking module and to determine atleast one user generated gesture based control command based on the userinput. The control module is further configured to determine a gazepoint area on the information presentation area where the user's gazepoint is located based on at least the gaze data signals and to executeat least one user action manipulating a view presented on the graphicalinformation presentation area based on the determined gaze point areaand at least one user generated gesture based control command, whereinthe user action is executed with the determined gaze point area as astarting point.

According to yet another aspect of the present invention, there isprovided a computer device associated with an information presentationarea. The computer device comprises input means adapted to detect usergenerated gestures and a gaze tracking module adapted to detect gazedata of a viewer of the information presentation area. The computerdevice further comprises a control module configured to acquire userinput from input means adapted to detect user generated gestures andgaze data signals from a gaze tracking module and to determine at leastone user generated gesture based control command based on the userinput. Moreover, the control module is configured to determine a gazepoint area on the information presentation area including the user'sgaze point based on at least the gaze data signals and to execute atleast one user action manipulating a view presented on the informationpresentation area based on the determined gaze point area and at leastone user generated gesture based control command, wherein the useraction is executed with the determined gaze point area as a startingpoint.

According to embodiments of the present invention, the computer devicemay, for example, be any one from the group of a personal computer,computer workstation, mainframe computer, a processor or device in avehicle, or a handheld device such as a cell phone, smartphone orsimilar device, portable music player (such as e.g. an iPod), laptopcomputers, computer games, electronic books, an iPAD or similar device,a Tablet, a Phoblet/Phablet.

According to embodiments of the present invention, the input means isconfigured to detect user gestures by a hand or a finger (or fingers),for example, relative a keyboard or an information presentation areausing, for example, an optical measurement technique or capacitivemeasurement technique.

According to an aspect of the present invention, there is provided asystem for user interaction with a wearable head mounted informationpresentation area. The system comprises input means configured as a gyroring adapted to detect user generated gestures and adapted to wirelesslycommunicate with a control module also communicatively connected to theinformation presentation area as well as a gaze tracking module adaptedto detect gaze data of a viewer of the information presentation area. Acontrol module configured to: acquire user input from the input meansand gaze data signals from the gaze tracking module; determine at leastone user generated gesture based control command based on the userinput; determine a gaze point area on the information presentation areaincluding the user's gaze point based on at least the gaze data signals;and execute at least one user action manipulating a view presented onthe graphical information presentation area based on the determined gazepoint area and at least one user generated gesture based controlcommand, wherein the user action is executed with the determined gazepoint area as a starting point.

According to a further aspect of the present invention, there isprovided a system for user interaction with an information presentationarea. The system comprises input means adapted to detect user generatedgestures, wherein the input means comprising at least one touchpadarranged on a steering device of a vehicle or adapted to be integratedin a steering device of a vehicle. Further, the system comprises a gazetracking module adapted to detect gaze data of a viewer of theinformation presentation area and a control module configured to:acquire user input from the input means and gaze data signals from thegaze tracking module; determine at least one user generated gesturebased control command based on the user input; determine a gaze pointarea on the information presentation area including the user's gazepoint based on at least the gaze data signals; and execute at least oneuser action manipulating a view presented on the graphical informationpresentation area based on the determined gaze point area and at leastone user generated gesture based control command, wherein the useraction is executed with the determined gaze point area as a startingpoint.

According to embodiments of the present invention, the input meansincludes a touchpad configured to enable a user to generate gesturebased control commands. The gesture based commands can for example begenerated by moving at least one finger over a surface of the touchpador touching a surface of the touchpad with, for example, the finger.

According to embodiments of the present invention, a dedicated part orarea of the touchpad surface is configured to receive gesture basedcontrol commands.

According to embodiments of the present invention, at least a firstdedicated part or area of the touchpad surface is configured to receivea first set of gesture based control commands and at least a second partor area of the touchpad surface is configured to receive a second set ofgesture based control commands. For example, the touchpad may beconfigured to receive gestures such as scrolling or zooming at adedicated area or part.

In embodiments of the present invention, the control module isconfigured to determine at least one gesture based control command basedon multiple simultaneous user input via the input means. Further, a gazepoint area on the information presentation area where the user's gazepoint is located is determined based on the gaze data signals and atleast one user action manipulating a view presented on the graphicalinformation presentation area is executed based on the determined gazepoint area and the at least one gesture based control command, whereinthe user action is executed with the determined gaze point area as astarting point.

According to embodiments of the present invention, an input module isconfigured to interpret signals representing at least one user generatedgesture to provide at least one gesture based control command reflectinga user's gesture. According to embodiments of the present invention, theinput module is arranged in the control module.

In embodiments of the present invention, the input module is configuredto interpret the signals representing the at least one user generatedgesture using gaze input signals and/or a predetermined set of possiblegesture based control commands, each possible control commandcorresponding to a particular user gesture relative the input means.

According to embodiments of the present invention, at least one objectis presented on the graphical information presentation area, the objectrepresenting at least one graphical user interface component andconfigured to be manipulated based on the user-generated gesture basedcontrol commands, wherein the control module is configured to determineif the gaze point of the user is on an object or in an area surroundingthat object based on the gaze data signals. Further, the control modulemay be configured to determine if the gaze point of the user has been onan object or in an area surrounding that object at a predetermined pointin time based on the gaze data signals. For example, the control modulemay be configured to determine if the gaze point of the user was on anobject or the area surrounding that object 0.1 seconds ago.

User activation of the object is enabled if the user's gaze point is onor within an area surrounding that object synchronized with a usergenerated activation command resulting from user input via the inputmeans, wherein the activated object can be manipulated by user generatedcommands resulting from user input via the input means. User activationof the object may also be enabled if the user's gaze point was on orwithin an area surrounding that object at the predetermined period oftime synchronized with a user generated activation command resultingfrom user input via the input means, wherein the activated object can bemanipulated by user generated commands resulting from user input via theinput means.

According to embodiments of the present invention, when the user touchesthe touchpad, the location of the initial gaze point is indicated by avisual feedback, such as a crosshairs or similar sign. The user mayadjust this initial location by moving the finger on the touchpad. Then,the user may, in a touchscreen like manner, interact with theinformation presentation area using different gestures. The strength ofthe visual feedback, e.g. the strength of the light of a crosshairs, maybe dependent on where the user's gaze is located on the informationpresentation area. For example, if a dragging operation to pan a windowis initiated at the gaze point, the visual feedback may initially bediscrete. When the dragging operation has been maintained for a period,the visual feedback can be strengthened to indicate for the user wherethe dragging operation is performed at the moment.

In the embodiments including a touchpad, the gestures are fingermovements relative the touchpad and each gesture is associated with orcorresponds to a particular gesture based control command resulting in auser action. Below, a non-exhaustive number of examples of user actionsthat can be executed using a combination of gestures and gaze arediscussed:

-   -   By gazing, for example, at an object presented on the        information presentation area and by, in connection to this,        pressing down and holding a finger on the touchpad during a        predetermined period of time, a visual feedback related to that        object is presented. For example, by pressing down and holding        the finger on the touchpad during a first period of time, the        object may be highlighted and, by continue to hold the finger on        the touchpad for a second period of time, an information box        presenting information regarding the object may be displayed.    -   By gazing, for example, at an object presented on the        information presentation area and by in connection to this        tapping on the touchpad using a finger, a primary action can be        initiated. For example, an application can be opened and started        by gazing at an icon representing the application and tapping on        the touchpad using a finger.    -   By gazing, for example, at an object presented on the        information presentation area and by, in connection to this,        lifting a finger (or fingers) that have been in contact with the        touchpad, a primary action can be initiated. For example, an        application can be opened and started by gazing at an icon        representing the application and lifting a finger (or fingers)        that have been in contact with the touchpad.    -   The user may slide or drag the view presented by the information        presentation area by gazing at the information presentation area        and by, in connection to this, sliding his or her finger over        the touchpad. The dragging is then initiated at the gaze point        of the user. A similar action to slide an object over the        information presentation area can be achieved by gazing at the        object and by, in connection to this, sliding the finger over        the touchpad. Both of these objectives may instead be        implemented in a way where two fingers are required to do the        swipe, or one finger is used for swiping while another finger        holds down a button.    -   The user may select an object for further actions by gazing at        the object and by, in connection to this, swiping his or her        finger downwards on the touchpad.    -   By gazing at an object or object part presented on the        information presentation area and by, in connection to this,        pinching with two of his or hers finger, it is possible to zoom        that object or object part. The same function can be implemented        also on a touchpad only able to sense single touch by having for        instance the thumb push a button or keyboard key and the finger        moving on the touchpad away from, or towards, the button or        keyboard key.    -   By gazing at an object or object part presented on the        information presentation area and by, in connection to this,        rotating with two of his or hers finger, it is possible to        rotate that object or object part. Similarly, when using a        touchpad only able to sense single touch the thumb can press a        button while a finger moves on the touchpad in a curve at a        constant distance from the button to rotate an object.    -   By gazing at an edge of the information presentation area and        sliding the finger over the touchpad in the direction that would        have been towards the centre of the information presentation        area if the gesture had been done at the gaze position, a menu        or other window hidden during normal use, such as a help menu,        can be presented or displayed. That is, a hidden menu or other        window can be displayed or presented if the user gazes at, for        example, the left edge of the information presentation area and        swipes his or her finger over the touchpad in the right        direction.    -   By gazing at a slider control, for example a volume control, the        finger can be moved up/down (or left/right for a horizontal        control) on the touch pad, on a predefined area of a touch        screen or above a keyboard to adjust the value of the slider        control.    -   By gazing at a checkbox control while doing a “check-gesture”        (such as a “V”) on the touchpad, the checkbox can be checked or        unchecked.    -   By gazing at a zoomable object or object part presented on the        information presentation area and while pressing hard on a        pressure sensitive touchpad with one finger (e.g. one of the        thumbs), it is possible to zoom in or out on said object using        the gaze point as the zoom center point, where each hard press        toggles between different zoom levels.    -   By gazing at an object or object part where several options are        available, for example “copy” or “rename”, the different options        can be displayed on different sides of the object after a preset        focusing dwell time has passed or after appropriate user input        has been provided. The touchpad or a predefined area of a touch        screen is thereafter used to choose action. For example, slide        left to copy and slide right to rename.

According to another embodiment of the present invention, the gazetracking module and the user input means are implemented in atouchscreen provided device such as an iPad or similar device. Thetouchscreen functions both as information presentation area and inputdevice for input of user gestures. A control module is included in thetouchscreen provided device and is configured to determine a gaze pointarea on the information presentation area, i.e. the touchscreen, wherethe user's gaze point is located based on the gaze data signals and toexecute at least one user action manipulating a view presented on thetouchscreen based on the determined gaze point area and at least oneuser generated gesture based control command, wherein the user action isexecuted with the determined gaze point area as a starting point. Theuser gestures are inputted via the touchscreen. According to thisembodiment, the user gestures, or finger movements on the touchscreen,are relative to the gaze point, which entails a more user friendly andergonomic use of touchscreen provided devices. For example, the user mayhold the device with both hands and interact with graphical userinterfaces on the touchscreen using the gaze and movement of the thumbs,where all user actions and activations have the gaze point of the useras starting point.

As mentioned, the gesture and gaze initiated actions discussed above areonly exemplary and there are a large number of further gestures incombination with gaze point resulting in an action that are conceivable.Below, some further examples are described:

-   -   Selection of an object or object part can be made by gazing at        that object or object part and pressing a finger (e.g. a thumb),        fine tuning by moving the finger and releasing the pressure        applied by the finger to select that object or object part;    -   Selection of an object or object part can be made by gazing at        that object or object part, pressing a finger (e.g. a thumb),        fine tuning by moving the finger, using another finger (e.g. the        other thumb) to tap for selecting that object or object part. In        addition, a double tap may be used for a “double click action”        and a quick downward movement may be used for a “right click”.    -   By gazing at a zoomable object or object part presented on the        information presentation area while moving a finger (e.g. one of        the thumbs) in a circular motion, it is possible to zoom in or        out of said object using the gaze point as the zoom center        point, where a clockwise motion performs a “zoom in” command and        a counterclockwise motion performs a “zoom out” command or vice        versa.    -   By gazing at a zoomable object or object part presented on the        information presentation area and in connection to this holding        one finger (e.g. one of the thumbs) still while moving another        finger (e.g. the other thumb) upwards or downwards, it is        possible to zoom in or out of said object using the gaze point        as the zoom center point, where an upwards motion performs a        “zoom in” command and a downwards motion performs a “zoom out”        command or vice versa.    -   By gazing at a zoomable object or object part presented on the        information presentation area while double-tapping on the touch        screen with one finger (e.g. one of the thumbs), it is possible        to zoom in or out of said object using the gaze point as the        zoom center point, where each double-tap toggles between        different zoom levels.    -   By gazing at a zoom able object or object part presented on the        information presentation area while sliding two fingers (e.g.        the two thumbs) simultaneously in opposite horizontal        directions, it is possible to zoom that object or object part.    -   By gazing at a zoomable object and in connection to this holding        a finger (e.g. one thumb) still on the touchscreen while moving        another finger (e.g. the other thumb) in a circular motion, it        is possible to zoom that object or object part.    -   By gazing at an object or object part presented on the        information presentation area and in connection to this holding        a finger (e.g. one of the thumbs) still on the touchscreen while        sliding another finger (e.g. the other thumb), it is possible to        slide or drag the view presented by the information presentation        area.    -   By gazing at an object or object part presented on the        information presentation area and in connection to this holding        a finger (e.g one of the thumbs) still on the touchscreen while        sliding another finger (e.g. the other thumb), it is possible to        slide or drag the view presented by the information presentation        area.    -   By gazing at an object or object part presented on the        information presentation area and while tapping or        double-tapping with a finger (e.g. one of the thumbs), an        automatic panning function can be activated so that the        presentation area is continuously slided from one of the edges        of the screen towards the center while the gaze point is near        the edge of the information presentation area, until a second        user input is received.    -   By gazing at an object or object part presented on the        information presentation area and while tapping or        double-tapping with a finger (e.g. one of the thumbs), the        presentation area is instantly slided according to the gaze        point (e.g. the gaze point is used to indicate the center of        where the information presentation area should be slided).    -   By gazing at a rotatable object or object part presented on the        information presentation area while sliding two fingers (e.g.        the two thumbs) simultaneously in opposite vertical directions,        it is possible to rotate that object or object part.

Before the two-finger gesture is performed, one of the fingers can beused to fine-tune the point of action. For example, a user feedbacksymbol like a “virtual finger” can be shown on the gaze point when theuser touches the touchscreen. The first finger can be used to slidearound to adjust the point of action relative to the original point.When the user touches the screen with the second finger, the point ofaction is fixed and the second finger is used for “clicking” on thepoint of action or for performing two-finger gestures like the rotate,drag and zoom examples above.

According to another embodiment of the current invention, the gazetracking module and the user input means are implemented in a portabledevice such as an iPad, ultrabook tablet or similar device. However,instead of performing the gestures with the thumbs on the presentationarea, one or two separate touchpads are placed on the back side of thedevice to allow two-finger gestures with other fingers than the thumb.

According to another embodiment of the current invention, the gazetracking module and the user input means are implemented in a vehicle.The information presentation area may be a heads-up display or aninfotainment screen. The input means may be one or two separate touchpads on the backside (for use with the index finger/s) or on the frontside (for use with the thumb/s) of the steering wheel.

According to another embodiment of the current invention, the gazetracking module and the information presentation area are implemented ina wearable head mounted display that may be designed to look as a pairof glasses (such as the solution described in U.S. Pat. No. 8,235,529).The user input means may include a gyro and be adapted to be worn on awrist, hand or at least one finger. For example the input means may be aring with a wireless connection to the glasses (or to a processing unitsuch as a smart phone that is communicatively connected to the glasses)and a gyro that detects small movements of the finger where the ring isworn. The detected movements representing gesture data may thenwirelessly be communicated to the glasses where gaze is detected andgesture based control commands based on the gesture data from the inputmeans is used to identify and execute user action.

Normally, in most applications, the touchpad is significantly smallerthan the information presentation area, which entails that in certainsituations the touchpad may impose limitations on the possible useractions. For example, it may be desired to drag or move an object overthe entire information presentation area while the user's movement of afinger or fingers is limited by the smaller touchpad area. Therefore, inembodiments of the present invention, a touchscreen like session can bemaintained despite that the user has removed the finger or fingers fromthe touchpad if, for example, a specific or dedicated button or keyboardkey is held down or pressed. Thereby, it is possible for the user toperform actions requiring multiple touches on the touchpad. For example,an object can be moved or dragged across the entire informationpresentation area by means of multiple dragging movements on thetouchpad.

In other embodiments of the present invention, a dragging movement onthe information presentation area or other user action is continuedafter the finger or fingers has reached an edge of the touchpad in thesame direction as the initial direction of the finger or fingers. Thecontinued movement or other actions may be continued until aninterruption command is delivered, which may be, for example, a pressingdown of a keyboard key or button, a tap on the touchpad or when thefinger or fingers is removed from the touchpad.

In further embodiments of the present invention, the speed of thedragging movement or other action is increased or accelerated when theuser's finger or fingers approaches the edge of the touchpad. The speedmay be decreased if the fingers or finger is moved in an oppositedirection.

In embodiments of the present invention, the action, e.g. a draggingmovement of an object, can be accelerated based on gaze position. Forexample, by gazing at an object, initiating a dragging operation of thatobject in a desired direction and thereafter gazing at a desired endposition for that object, the speed of the object movement will behigher the longer the distance between the initial position of theobject and the desired end position is.

In other embodiments of the present invention voice commands may be usedto choose what action to perform on the object currently being gazed atand then a gesture is required to fulfill the action. For instance avoice command such as the word “move” may allow the user to move theobject currently being gazed at by moving a finger over the touchpad ortouchscreen. Another action to perform may be to delete an object. Inthis case the word “delete” may allow deletion of the object currentlybeing gazed at, but additionally a gesture, such as swiping downwards isrequired to actually delete the object. Thus, the object to act on ischosen by gazing at it, the specific action to perform is chosen by avoice command and the movement to perform or the confirmation is done bya gesture.

Further objects and advantages of the present invention will bediscussed below by means of exemplifying embodiments.

These and other features, aspects and advantages of the invention willbe more fully understood when considered with respect to the followingdetailed description, appended claims and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings are not necessarily drawn to scale and illustrategenerally, by way of example, but no way of limitation, variousembodiments of the present invention. Thus, exemplifying embodiments ofthe invention are illustrated by way of example and not by way oflimitation in the figures of the accompanying drawings in which likereferences indicate similar elements. It should be noted that referencesto “an” or “one” embodiment in this discussion are not necessarily tothe same embodiment, and such references mean at least one.

FIG. 1 shows an overview picture of a user controlling a computerapparatus in which the present invention is implemented;

FIG. 2 is a block diagram illustrating an embodiment of an arrangementin accordance with the present invention;

FIG. 3 is a block diagram illustrating another embodiment of anarrangement in accordance with the present invention;

FIG. 4 illustrates an exemplary gesture resulting in a user generatedgesture based control command in accordance with the present invention;

FIG. 5 illustrates another exemplary gesture resulting in a usergenerated gesture based control command in accordance with the presentinvention;

FIG. 6 illustrates a further exemplary gesture resulting in a usergenerated gesture based control command in accordance with the presentinvention;

FIG. 7 illustrates yet another exemplary gesture resulting in a usergenerated gesture based control command in accordance with the presentinvention;

FIG. 8 illustrates a further exemplary gesture resulting in a usergenerated gesture based control command in accordance with the presentinvention;

FIG. 9 illustrates another exemplary gesture resulting in a usergenerated gesture based control command in accordance with the presentinvention;

FIG. 10 illustrates yet another exemplary gesture resulting in a usergenerated gesture based control command in accordance with the presentinvention;

FIG. 11a shows an overview picture of a touchscreen provided device inwhich a further embodiment of the present invention is implemented;

FIG. 11b shows an overview picture of a device provided with touchpadson a backside in which a further embodiment of the present invention isimplemented;

FIG. 12 is a block diagram illustrating the embodiment in accordancewith the present invention shown in FIG. 11 a;

FIG. 13a is a schematic view of a control module according to anembodiment of the present invention;

FIG. 13b is a schematic view of a control module according to anotherembodiment of the present invention;

FIG. 13c is a schematic view of a control module according to anotherembodiment of the present invention;

FIG. 14 is a schematic view of a wireless transmit/receive unit, WTRU,according to an embodiment of the present invention;

FIG. 15a is a schematic view of an embodiment of a computer device orhandheld device in accordance with an embodiment of the presentinvention;

FIG. 15b is a schematic view of another embodiment of a computer deviceor handheld device in accordance with the present invention;

FIG. 16 is a schematic flow chart illustrating steps of an embodiment ofa method in accordance with an embodiment of the present invention;

FIG. 17 is a schematic flow chart illustrating steps of anotherembodiment of a method in accordance with the present invention; and

FIG. 18 is a schematic flow chart illustrating steps of a furtherembodiment of a method in accordance with an embodiment of the presentinvention;

FIG. 19 is a schematic flow chart illustrating steps of anotherembodiment of a method in accordance with an embodiment of the presentinvention;

FIG. 20 is a block diagram illustrating a further embodiment of anarrangement in accordance with the present invention;

FIG. 21 is a schematic illustration of yet another implementation of thepresent invention;

FIG. 22 is a schematic illustration of a further implementation of thepresent invention; and

FIG. 23 is a schematic illustration of an implementation of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

As used herein, the term “module” refers to an application specificintegrated circuit (ASIC), an electronic circuit, a processor (shared,dedicated, or group) and memory that execute one or more softwareprograms, a combinational logic circuit, or other suitable componentsthat provide the described functionality. The term “module” furtherrefers to a specific form of software necessary to practice the methodsdescribed herein and particularly the functions described in connectionwith each specific “module”. It is believed that the particular form ofsoftware will be determined primarily by the particular systemarchitecture employed in the system and by the particular methodologiesemployed by the system according to the present invention.

The following is a description of exemplifying embodiments in accordancewith the present invention. This description is not to be taken inlimiting sense, but is made merely for the purposes of describing thegeneral principles of the invention. It is to be understood that otherembodiments may be utilized and structural and logical changes may bemade without departing from the scope of the present invention.

With reference first to FIGS. 1, 2, 3 and 20, embodiments of a computersystem according to the present invention will be described. FIG. 1shows an embodiment of a computer system with integrated gaze and manualcontrol according to the present invention. The user 110 is able tocontrol the computer system 10 at least partly based on an eye-trackingsignal D_(EYE), which described the user's point of regard x, y on ainformation presentation area or display 20 and based on user generatedgestures, i.e. a movement of at least one body part of the user can bedetected, generating gesture based control commands via user input means50 such as a touchpad 51.

In the context of the present invention, as mentioned above, the term“touchpad” (or the term “trackpad”) refers to a pointing devicefeaturing a tactile sensor, a specialized surface that can translate themotion and position of a user's fingers to a relative position on ascreen (information presentation area). Touchpads are a common featureof laptop computers, and are also used as a substitute for a mouse wheredesk space is scarce. Because they vary in size, they can also be foundon personal digital assistants (PDAs) and some portable media players.Wireless touchpads are also available as detached accessories. Touchpadsoperate in one of several ways, including capacitive sensing andconductance sensing. The most common technology used today entailssensing the capacitive virtual ground effect of a finger, or thecapacitance between sensors. While touchpads, like touchscreens, areable to sense absolute position, resolution is limited by their size.For common use as a pointer device, the dragging motion of a finger istranslated into a finer, relative motion of the cursor on the screen,analogous to the handling of a mouse that is lifted and put back on asurface. Hardware buttons equivalent to a standard mouse's left andright buttons are positioned below, above, or beside the touchpad.Netbooks sometimes employ the last as a way to save space. Sometouchpads and associated device driver software may interpret tappingthe pad as a click, and a tap followed by a continuous pointing motion(a “click-and-a-half”) can indicate dragging. Tactile touchpads allowfor clicking and dragging by incorporating button functionality into thesurface of the touchpad itself. To select, one presses down on thetouchpad instead of a physical button. To drag, instead performing the“click-and-a-half” technique, one presses down while on the object,drags without releasing pressure and lets go when done. Touchpad driverscan also allow the use of multiple fingers to facilitate the other mousebuttons (commonly two-finger tapping for the center button). Sometouchpads have “hotspots”, locations on the touchpad used forfunctionality beyond a mouse. For example, on certain touchpads, movingthe finger along an edge of the touch pad will act as a scroll wheel,controlling the scrollbar and scrolling the window that has the focusvertically or horizontally. Apple uses two-finger dragging for scrollingon their trackpads. Also, some touchpad drivers support tap zones,regions where a tap will execute a function, for example, pausing amedia player or launching an application. All of these functions areimplemented in the touchpad device driver software, and can be disabled.Touchpads are primarily used in self-contained portable laptop computersand do not require a flat surface near the machine. The touchpad isclose to the keyboard, and only very short finger movements are requiredto move the cursor across the display screen; while advantageous, thisalso makes it possible for a user's thumb to move the mouse cursoraccidentally while typing. Touchpad functionality is available fordesktop computers in keyboards with built-in touchpads.

Examples of touchpads include one-dimensional touchpads used as theprimary control interface for menu navigation on second-generation andlater iPod Classic portable music players, where they are referred to as“click wheels”, since they only sense motion along one axis, which iswrapped around like a wheel. In another implementation of touchpads, thesecond-generation Microsoft Zune product line (the Zune 80/120 and Zune4/8) uses touch for the Zune Pad. Apple's PowerBook 500 series was itsfirst laptop to carry such a device, which Apple refers to as a“trackpad”. Apple's more recent laptops feature trackpads that can senseup to five fingers simultaneously, providing more options for input,such as the ability to bring up the context menu by tapping two fingers.In late 2008 Apple's revisions of the MacBook and MacBook Proincorporated a “Tactile Touchpad” design with button functionalityincorporated into the tracking surface.

The present invention provides a solution enabling a user of a computersystem without a traditional touchscreen to interact with graphical userinterfaces in a touchscreen like manner using a combination of gazebased input and gesture based user commands. Furthermore, the presentinvention offers a solution for touchscreen like interaction using gazeinput and gesture based input as a complement or an alternative totouchscreen interactions with a computer device having a touchscreen.

The display 20 may hence be any type of known computer screen ormonitor, as well as combinations of two or more separate displays. Forexample, the display 20 may constitute a regular computer screen, astereoscopic screen, a heads-up display (HUD) in a vehicle, or at leastone head-mounted display (HMD).

The computer 30 may, for example, be any one from the group of apersonal computer, computer workstation, mainframe computer, a processorin a vehicle, or a handheld device such as a cell phone, portable musicplayer (such as e.g. an iPod), laptop computers, computer games,electronic books and similar other devices. The present invention mayalso be implemented in “intelligent environment” where, for example,objects presented on multiple displays can be selected and activated.

In order to produce the gaze tracking signal D_(EYE), a gaze trackerunit 40 is included in the display 20, or is associated with the display20. A suitable gaze tracker is described in the U.S. Pat. No. 7,572,008,titled “Method and Installation for detecting and following an eye andthe gaze direction thereof”, by the same applicant, which hereby isincorporated in its entirety.

The software program or software implemented instructions associatedwith the gaze tracking module 40 may be included within the gazetracking module 40. The specific example shown in FIGS. 2, 3 and 20illustrates the associated software implemented in a gaze trackingmodule, which may be included solely in the computer 30, in the gazetracking module 40, or in a combination of the two, depending on theparticular application.

The computer system 10 comprises a computer device 30, a gaze trackingmodule 40, a display 20, a control module 36, 36′ and user input means50, 50′ as shown in FIGS. 2, 3 and 20. The computer device 30 comprisesseveral other components in addition to those illustrated in FIGS. 2 and20 but these components are omitted from FIGS. 2, 3 and 20 inillustrative purposes.

The user input means 50, 50′ comprises elements that are sensitive topressure, physical contact, gestures, or other manual control by theuser, for example, a touchpad 51. Further, the input device means 50,50′ may also include a computer keyboard, a mouse, a “track ball”, orany other device, for example, an IR-sensor, voice activated inputmeans, or a detection device of body gestures or proximity based inputcan be used. However, in the specific embodiments shown in FIGS. 2, 3and 20, a touchpad 51 is included in the user input device 50, 50′.

An input module 32, which may be a software module included solely in acontrol module 36′ or in the user input means 50 or as a module separatefrom the control module and the input means 50′, is configured toreceive signals from the touchpad 51 reflecting a user's gestures.Further, the input module 32 is also adapted to interpret the receivedsignals and provide, based on the interpreted signals, gesture basedcontrol commands, for example, a tap command to activate an object, aswipe command or a slide command.

If the input module 32 is included in the input means 50, gesture basedcontrol commands are provided to the control module 36, see FIG. 2. Inembodiments of the present invention, the control module 36′ includesthe input module 32 based on gesture data from the user input means 50′,see FIG. 3.

The control module 36, 36′ is further configured to acquire gaze datasignals from the gaze tracking module 40. Further, the control module36, 36′ is configured to determine a gaze point area 120 on theinformation presentation area 20 where the user's gaze point is locatedbased on the gaze data signals. The gaze point area 120 is preferably,as illustrated in FIG. 1, a local area around a gaze point of the user.

Moreover, the control module 36, 36′ is configured to execute at leastone user action manipulating a view presented on the graphicalinformation presentation area 20 based on the determined gaze point areaand the at least one user generated gesture based control command,wherein the user action is executed with the determined gaze point areaas a starting point.

The control module 36, 36′ may be integrated in the computer device 30or may be associated or coupled to the computer device 30.

Hence, the present invention allows a user to interact with a computerdevice 30 in touchscreen like manner, e.g. manipulate objects presentedon the information presentation area 20, using gaze and gestures, e.g.by moving at least one finger on a touchpad 51.

Preferably, when the user touches the touchpad 51, the location of theinitial gaze point is indicated by a visual feedback, such as acrosshairs or similar sign. This initial location can be adjusted bymoving the finger on the touchpad 51. Thereafter, the user can, in atouchscreen like manner, interact with the information presentation area20 using different gestures and the gaze. In the embodiment including atouchpad, the gestures are finger movements relative the touchpad 51 andeach gesture is associated with or corresponds to particular gesturebased user command resulting in a user action.

Below, a non-exhaustive number of examples of user actions that can beexecuted using a combination of gestures and gaze will be discussed withregard to FIG. 4-10:

-   -   By gazing, for example, at an object presented on the        information presentation area 20 and by in connection to this,        touching the touchpad or pressing down and holding a finger 60        (see FIG. 4) on the touchpad 51 during a period of y ms, that        object is highlighted. If the finger 60 is held down during a        second period of z ms, an information box may be displayed        presenting information regarding that object. In FIG. 4, this        gesture is illustrated in relation to a touchpad 51.    -   By gazing, for example, at an object presented on the        information presentation area 20 and by in connection to this        tapping on the touchpad 51 using a finger 71, a primary action        can be initiated. For example, an application can be opened and        started by gazing at an icon representing the application and        tapping on the touchpad 51 using a finger. In FIG. 5, this        gesture is illustrated in relation to a touchpad 51.    -   The user may slide or drag the view presented by the information        presentation area 20 by gazing somewhere on the information        presentation area 20 and by, in connection to this, sliding his        or her finger 81 over the touchpad 51. A similar action to slide        an object over the information presentation area 20 can be        achieved by gazing at the object and by, in connection to this,        sliding the finger 81 over the touchpad 51. This gesture is        illustrated in FIG. 6 in relation to the touchpad 51. Of course,        this gesture can be executed by means of more than one finger,        for example, by using two fingers.    -   The user may select an object for further actions by gazing at        the object and by, in connection to this, swiping his or her        finger 91 on the touchpad 51 in a specific direction. This        gesture is illustrated in FIG. 7 in relation to the touchpad 51.        Of course, this gesture can be executed by means of more than        one finger, for example, by using two fingers.    -   By gazing at an object or object part presented on the        information presentation area 20 and by, in connection to this,        pinching with two of his or hers finger 101 and 102, it is        possible to zoom out that object or object part. This gesture is        illustrated in FIG. 8 in relation to the touchpad 51. Similarly,        by gazing at an object or object part presented on the        information presentation area 20 and by, in connection to this,        moving the fingers 101 and 102 apart, it is possible to expand        or zoom in that object or object part.    -   By gazing at an object or object part presented on the        information presentation area 20 and by, in connection to this,        rotating with two of his or hers finger 111 and 112, it is        possible to rotate that object or object part. This gesture is        illustrated in FIG. 9 in relation to the touchpad 51.    -   By gazing at an edge or frame part of the information        presentation area 20 or at an area in proximity to the edge or        frame and, in connection to this, sliding his or her finger or        fingers 124 on the touchpad 51 in a direction which if performed        at the point of gaze would have been from the edge towards a        centre of the information presentation area a menu may come in        from the edge.    -   By gazing at a slider control, for example a volume control, the        finger can be moved up/down (or left/right for a horizontal        control) to adjust the value of the slider control. With        appropriate input means this gesture can be detected on a        touchpad, on a touch screen or in air without physically        touching the input means.    -   By gazing at a checkbox control while doing a “check-gesture”        (such as a “V”) on the touchpad, the checkbox can be checked or        unchecked. With appropriate input means this gesture can be        detected on a touchpad, on a touch screen or in air without        physically touching the input means.    -   By gazing at an object or object part where several options are        available, for example “copy” or “rename”, the different options        can be displayed on different sides of the object after a preset        focusing dwell time has passed or after appropriate user input        has been provided. Thereafter a gesture is done to choose        action. For example, swipe left to copy and swipe right to        rename. With appropriate input means this gesture can be        detected on a touchpad, on a touch screen or in air without        physically touching the input means.

By pressing the finger harder on the touchpad, i.e. increasing thepressure of a finger touching the touchpad, a sliding mode can beinitiated. For example, by gazing at an object, touching the touchpad,increasing the pressure on the touchpad and moving the finger or fingerover the touchscreen, the object can be moved or dragged over theinformation presentation area. When the user removes the finger from thetouchpad 51, the touchscreen like session is finished. The user maythereafter start a new touchscreen like session by gazing at theinformation presentation area 20 and placing the finger on the touchpad51.

As mentioned, the gesture and gaze initiated actions discussed above areonly exemplary and there are a large number of further gestures incombination with gaze point resulting in an action that are conceivable.With appropriate input means many of these gestures can be detected on atouchpad, on a predefined area of a touch screen, in air withoutphysically touching the input means, or by an input means worn on afinger or a hand of the user. Below, some further examples aredescribed:

-   -   Selection of an object or object part can be made by gazing at        that object or object part and pressing a finger (e.g. a thumb),        fine tuning by moving the finger and releasing the pressure        applied by the finger to select that object or object part;    -   Selection of an object or object part can be made by gazing at        that object or object part, pressing a finger (e.g. a thumb),        fine tuning by moving the finger, using another finger (e.g. the        other thumb) to tap for selecting that object or object part. In        addition, a double tap may be used for a “double click action”        and a quick downward movement may be used for a “right click”.    -   By gazing at a zoomable object or object part presented on the        information presentation area while moving a finger (e.g. one of        the thumbs) in a circular motion, it is possible to zoom in or        out of the said object using the gaze point as the zoom center        point, where a clockwise motion performs a “zoom in” command and        a counterclockwise motion performs a “zoom out” command or vice        versa.    -   By gazing at a zoomable object or object part presented on the        information presentation area and in connection to this holding        one finger (e.g. one of the thumbs) still while moving another        finger (e.g. the other thumb) upwards and downwards, it is        possible to zoom in or out of the said object using the gaze        point as the zoom center point, where an upwards motion performs        a “zoom in” command and a downwards motion performs a “zoom out”        command or vice versa.    -   By gazing at a zoomable object or object part presented on the        information presentation area and while pressing hard on a        pressure-sensitive touchpad with one finger (e.g. one of the        thumbs), it is possible to zoom in or out on the said object        using the gaze point as the zoom center point, where each hard        press toggles between different zoom levels.    -   By gazing at a zoomable object or object part presented on the        information presentation area while double-tapping on a touchpad        with one finger (e.g. one of the thumbs), it is possible to zoom        in or out of the said object using the gaze point as the zoom        center point, where each double-tap toggles between different        zoom levels.    -   By gazing at a zoomable object or object part presented on the        information presentation area while sliding two fingers (e.g.        the two thumbs) simultaneously in opposite horizontal        directions, it is possible to zoom that object or object part.    -   By gazing at a zoomable object and in connection to this holding        finger (e.g. one thumb) still on the touchscreen while moving        another finger (e.g. the other thumb) in a circular motion, it        is possible to zoom that object or object part.    -   By gazing at an object or object part presented on the        information presentation area and in connection to this holding        a finger (e.g one of the thumbs) still on the touchscreen while        sliding another finger (e.g. the other thumb), it is possible to        slide or drag the view presented by the information presentation        area.    -   By gazing at an object or object part presented on the        information presentation area and in connection to this holding        a finger (e.g one of the thumbs) still on the touchscreen while        sliding another finger (e.g. the other thumb), it is possible to        slide or drag the view presented by the information presentation        area.    -   By gazing at an object or object part presented on the        information presentation area and while tapping or        double-tapping with a finger (e.g. one of the thumbs), an        automatic panning function can be activated so that the        presentation area is continuously slided from one of the edges        of the screen towards the center while the gaze point is near        the edge of the information presentation area, until a second        user input is received.    -   By gazing at an object or object part presented on the        information presentation area and while tapping or        double-tapping with a finger (e.g. one of the thumbs), the        presentation area is instantly slided according to the gaze        point (e.g. the gaze point is used to indicate the center of        where the information presentation area should be slided).    -   By gazing at a rotatable object or object part presented on the        information presentation area while sliding two fingers (e.g.        the two thumbs) simultaneously in opposite vertical directions,        it is possible to rotate that object or object part.

Before the two-finger gesture is performed, one of the fingers can beused to fine-tune the point of action. For example, a user feedbacksymbol like a “virtual finger” can be shown on the gaze point when theuser touches the touchscreen. The first finger can be used to slidearound to adjust the point of action relative to the original point.When user touches the screen with the second finger, the point of actionis fixed and the second finger is used for “clicking” on the point ofaction or for performing two-finger gestures like the rotate, drag andzoom examples above.

In embodiments of the present invention, the touchscreen like sessioncan be maintained despite that the user has removed the finger orfingers from the touchpad if, for example, a specific or dedicatedbutton or keyboard key is held down or pressed. Thereby, it is possiblefor the user to perform actions requiring multiple touches on thetouchpad. For example, an object can be moved or dragged across theentire information presentation area by means of multiple draggingmovements on the touchpad.

With reference now to FIGS. 11a, 11b and 12, further embodiments of thepresent invention will be discussed. FIG. 11a shows a further embodimentof a system with integrated gaze and manual control according to thepresent invention. This embodiment of the system is implemented in adevice 100 with a touchscreen 151 such as an iPad or similar device. Theuser is able to control the device 100 at least partly based on gazetracking signals which describes the user's point of regard x, y on thetouchscreen 151 and based on user generated gestures, i.e. a movement ofat least one body part of the user can be detected, generating gesturebased control commands via user input means 150 including thetouchscreen 151.

The present invention provides a solution enabling a user of a device100 with a touchscreen 151 to interact with a graphical user interfacesusing gaze as direct input and gesture based user commands as relativeinput. Thereby, it is possible, for example, to hold the device 100 withboth hands and interact with a graphical user interface 180 presented onthe touchscreen with gaze and the thumbs 161 and 162 as shown in FIG. 11a.

In an alternative embodiment, one or more touchpads 168 can be arrangedon the backside of the device 100′, i.e. on the side of the device onwhich the user normally do not look at during use. This embodiment isillustrated in FIG. 11b . Thereby, a user is allowed to control thedevice at least partly based on gaze tracking signals which describesthe user's point of regard x, y on the information presentation area andbased on user generated gestures, i.e. a movement of at least one fingeron the one or more touchpads 168 on the backside of the device 100′,generating gesture based control commands interpreted by the controlmodule. In order to produce the gaze tracking signal, a gaze trackingmodule 140 is included in the device 100, 100′. A suitable gaze trackeris described in the U.S. Pat. No. 7,572,008, titled “Method andInstallation for detecting and following an eye and the gaze directionthereof”, by the same applicant, which hereby is incorporated in itsentirety.

The software program or software implemented instructions associatedwith the gaze tracking module 140 may be included within the gazetracking module 140.

The device 100 comprises a gaze tracking module 140, user input means150 including the touchscreen 151 and an input module 132, and a controlmodule 136 as shown in FIG. 12. The device 100 comprises several othercomponents in addition to those illustrated in FIG. 12 but thesecomponents are omitted from FIG. 12 in illustrative purposes.

The input module 132, which may be a software module included solely ina control module or in the user input means 150, is configured toreceive signals from the touchscreen 151 reflecting a user's gestures.Further, the input module 132 is also adapted to interpret the receivedsignals and provide, based on the interpreted signals, gesture basedcontrol commands, for example, a tap command to activate an object, aswipe command or a slide command.

The control module 136 is configured to acquire gaze data signals fromthe gaze tracking module 140 and gesture based control commands from theinput module 132. Further, the control module 136 is configured todetermine a gaze point area 180 on the information presentation area,i.e. the touchscreen 151, where the user's gaze point is located basedon the gaze data signals. The gaze point area 180 is preferably, asillustrated in FIG. 1, a local area around a gaze point of the user.

Moreover, the control module 136 is configured to execute at least oneuser action manipulating a view presented on the touchscreen 151 basedon the determined gaze point area and the at least one user generatedgesture based control command, wherein the user action is executed withthe determined gaze point area as a starting point. All user actionsdescribed in the context of this application may also be executed withthis embodiment of the present invention.

In a possible further embodiment, when the user touches the touchscreen151, the location of the initial gaze point is indicated by a visualfeedback, such as a crosshairs or similar sign. This initial locationcan be adjusted by moving the finger on the touchscreen 151, forexample, using a thumb 161 or 162. Thereafter, the user can interactwith the touchscreen 151 using different gestures and the gaze, wherethe gaze is the direct indicator of the user's interest and the gesturesare relative to the touchscreen 151. In the embodiment including atouchscreen, the gestures are finger movements relative the touchscreen151 and each gesture is associated with or corresponds to particulargesture based user command resulting in a user action.

With reference now to FIGS. 13a, 13b and 13c , control modules forgenerating gesture based commands during user interaction with aninformation presentation area 201, for example, associated with a WTRU(described below with reference to FIG. 14), or a computer device orhandheld portable device (described below with reference to FIG. 15a or15 b), or in a vehicle (described below with reference to FIG. 21), orin a wearable head mounted display (described below with reference toFIG. 22) will be described. Parts or modules described above will not bedescribed in detail again in connection to this embodiment.

According to an embodiment of the present invention shown in FIG. 13a ,the control module 200 is configured to acquire user input from inputmeans 205, for example, included in a device in which the control modulemay be arranged in, adapted to detect user generated gestures. For thispurpose, the control module 200 may include an input module 232comprising a data acquisition module 210 configured to translate thegesture data from the input means 205 into an input signal. The inputmeans 205 may include elements that are sensitive to pressure, physicalcontact, gestures, or other manual control by the user, for example, atouchpad. Further, the input means 205 may also include a computerkeyboard, a mouse, a “track ball”, or any other device, for example, anIR-sensor, voice activated input means, or a detection device of bodygestures or proximity based input can be used.

Further, the input module 232 is configured to determine at least oneuser generated gesture based control command based on the input signal.For this purpose, the input module 232 further comprises a gesturedetermining module 220 communicating with the data acquisition module210. The gesture determining module 220 may also communicate with thegaze data analyzing module 240. The gesture determining module 220 maybe configured to check whether the input signal corresponds to apredefined or predetermined relative gesture and optionally use gazeinput signals to interpret the input signal. For example, the controlmodule 200 may comprise a gesture storage unit (not shown) storing alibrary or list of predefined gestures, each predefined gesturecorresponding to a specific input signal. Thus, the gesture determiningmodule 220 is adapted to interpret the received signals and provide,based on the interpreted signals, gesture based control commands, forexample, a tap command to activate an object, a swipe command or a slidecommand.

A gaze data analyzing module 240 is configured to determine a gaze pointarea on the information presentation area 201 including the user's gazepoint based on at least the gaze data signals from the gaze trackingmodule 235. The information presentation area 201 may be a display ofany type of known computer screen or monitor, as well as combinations oftwo or more separate displays, which will depend on the specific deviceor system in which the control module is implemented in. For example,the display 201 may constitute a regular computer screen, a stereoscopicscreen, a heads-up display (HUD) in a vehicle, or at least onehead-mounted display (HMD). Then, a processing module 250 may beconfigured to execute at least one user action manipulating a viewpresented on the information presentation area 201 based on thedetermined gaze point area and at least one user generated gesture basedcontrol command, wherein the user action is executed with the determinedgaze point area as a starting point. Hence, the user is able to controla device or system at least partly based on an eye-tracking signal whichdescribed the user's point of regard x, y on the informationpresentation area or display 201 and based on user generated gestures,i.e. a movement of at least one body part of the user can be detected,generating gesture based control commands via user input means 205 suchas a touchpad.

According to another embodiment a control module according to thepresent invention shown in FIG. 13b , the control module 260 isconfigured to acquire gesture based control commands from an inputmodule 232′. The input module 232′ may comprise a gesture determiningmodule and a data acquisition module as described above with referenceto FIG. 13a . A gaze data analyzing module 240 is configured todetermine a gaze point area on the information presentation area 201including the user's gaze point based on at least the gaze data signalsreceived from the gaze tracking module 235. The information presentationarea 201 may be a display of any type of known computer screen ormonitor, as well as combinations of two or more separate displays, whichwill depend on the specific device or system in which the control moduleis implemented in. For example, the display 201 may constitute a regularcomputer screen, a stereoscopic screen, a heads-up display (HUD) in avehicle, or at least one head-mounted display (HMD). A processing module250 may be configured to execute at least one user action manipulating aview presented on the information presentation area 201 based on thedetermined gaze point area and at least one user generated gesture basedcontrol command, wherein the user action is executed with the determinedgaze point area as a starting point. Hence, the user is able to controla device or system at least partly based on an eye-tracking signal whichdescribed the user's point of regard x, y on the informationpresentation area or display 201 and based on user generated gestures,i.e. a movement of at least one body part of the user can be detected,generating gesture based control commands via user input means 205 suchas a touchpad.

With reference to FIG. 13c , a further embodiment of a control moduleaccording to the present invention will be discussed. The input module232″ is distributed such that the data acquisition module 210 isprovided outside the control module 280 and the gesture determiningmodule 220 is provided in the control module 280. A gaze data analyzingmodule 240 is configured to determine a gaze point area on theinformation presentation area 201 including the user's gaze point basedon at least the gaze data signals received from the gaze tracking module235. The information presentation area 201 may be a display of any typeof known computer screen or monitor, as well as combinations of two ormore separate displays, which will depend on the specific device orsystem in which the control module is implemented in. For example, thedisplay 201 may constitute a regular computer screen, a stereoscopicscreen, a heads-up display (HUD) in a vehicle, or at least onehead-mounted display (HMD). A processing module 250 may be configured toexecute at least one user action manipulating a view presented on theinformation presentation area 201 based on the determined gaze pointarea and at least one user generated gesture based control command,wherein the user action is executed with the determined gaze point areaas a starting point. Hence, the user is able to control a device orsystem at least partly based on an eye-tracking signal which describedthe user's point of regard x, y on the information presentation area ordisplay 201 and based on user generated gestures, i.e. a movement of atleast one body part of the user can be detected, generating gesturebased control commands via user input means 205 such as a touchpad.

With reference to FIG. 14, a wireless transmit/receive unit (WTRU) suchas a cellular telephone or a smartphone, in accordance with the presentinvention will be described. Parts or modules described above will notbe described in detail again. Further, only parts or modules related tothe present invention will be described below. Accordingly, the WTRUincludes a large number of additional parts, units and modules that arenot described herein such as antennas and transmit/receive units. Thewireless transmit/receive unit (WTRU) 300 is associated with aninformation presentation area 301 and further comprises input means 305,including e.g. an input module as has been described above, adapted todetect user generated gestures and a gaze tracking module 325 adapted todetect gaze data of a viewer of the information presentation area 301.The WTRU further comprises a control module 200, 260 or 280 as describedabove with reference to FIGS. 13a, 13b and 13c . The user is able tocontrol the WTRU at least partly based on an eye-tracking signal whichdescribes the user's point of regard x, y on the informationpresentation area or display 301 and based on user generated gestures,i.e. a movement of at least one body part of the user can be detected,generating gesture based control commands via user input means 305 suchas a touchpad. All user actions described in the context of thisapplication may also be executed with this embodiment of the presentinvention.

With reference to FIGS. 15a and 15b , a computer device or handheldportable device in accordance with the present invention will bedescribed. Parts or modules described above will not be described indetail again. Further, only parts or modules related to the presentinvention will be described below. Accordingly, the device includes alarge number of additional parts, units and modules that are notdescribed herein such as memory units (e.g. RAM/ROM), or processingunits. The computer device or handheld portable device 400 may, forexample, be any one from the group of a personal computer, computerworkstation, mainframe computer, a processor or device in a vehicle, ora handheld device such as a cell phone, smartphone or similar device,portable music player (such as e.g. an iPod), laptop computers, computergames, electronic books, an iPAD or similar device, a Tablet, aPhoblet/Phablet.

The computer device or handheld device 400 a is connectable to aninformation presentation area 401 a (e.g. an external display or aheads-up display (HUD), or at least one head-mounted display (HMD)), asshown in FIG. 15a , or the computer device or handheld device 400 bincludes an information presentation area 401 b, as shown in FIG. 15b ,such as a regular computer screen, a stereoscopic screen, a heads-updisplay (HUD), or at least one head-mounted display (HMD). Furthermore,the computer device or handheld device 400 a, 400 b comprises inputmeans 405 adapted to detect user generated gestures and a gaze trackingmodule 435 adapted to detect gaze data of a viewer of the informationpresentation area 401. Moreover, the computer device or handheld device400 a, 400 b comprises a control module 200, 260, or 280 as describedabove with reference to FIG. 13a, 13b or 13 c. The user is able tocontrol the computer device or handheld device 400 a, 400 b at leastpartly based on an eye-tracking signal which described the user's pointof regard x, y on the information presentation area or display 401 andbased on user generated gestures, i.e. a movement of at least one bodypart of the user can be detected, generating gesture based controlcommands via user input means 405 such as a touchpad. All user actionsdescribed in the context of this application may also be executed withthis embodiment of the present invention.

With reference now to FIG. 16-19, example embodiments of methodsaccording to the present invention will be described. The methodembodiments described in connection with FIGS. 16-19 are implemented inan environment where certain steps are performed in a device, e.g. aWTRU described above with reference to FIG. 14, or a computer device orhandheld device described above with reference to FIG. 15a or 15 b andcertain steps are performed in a control module, e.g. a control moduleas described above with reference to FIGS. 13a, 13b and 13c . As theskilled person realizes, the methods described herein can also beimplemented in other environments, as, for example, in a system asdescribed above with reference to FIGS. 2, 3 and 20 or inimplementations illustrated in FIGS. 21-23. Similar or like stepsperformed in the different embodiments will be denoted with the samereference numeral hereinafter.

With reference first to FIG. 16, the device is waiting for user input instep S500. In step S510, the user touches a touch sensitive area on thedevice (e.g. input means as described above) with one or more fingers ofeach hand. This step is not a part of the method according toembodiments of the invention. There are a large number of conceivablegestures that the user can use to control actions of the device, and anon-exhaustive number of such gestures have been described above. Atstep S520, the gesture data, i.e. the user input, is translated into aninput signal. At step S530, it is checked whether the input signalcorresponds to a predefined or predetermined relative gesture. If not,the procedure returns back to step S500. On the other hand, if yes (i.e.the input signal corresponds to a predefined gesture), a gesture basedcontrol command is generated at step S570. At step S540, the user looksat a screen or an information presentation area and at step S550 theuser's gaze is detected at the information presentation area. The stepS540 is not a part of the method according to embodiments of the presentinvention. In step S560, a gaze point area including a user's point ofgaze on the screen or information presentation area. At step S580, anaction corresponding to the relative gesture at the user's point of gazeis performed based on the gesture based control command and thedetermined gaze point at the information presentation area.

With reference to FIG. 17, the device is waiting for user input in stepS500. In step S590, the user makes a gesture with one or more fingersand/or at least one hand in front of the information presentation area(which gesture is interpreted by input means as described above). Thestep S590 is not a part of the method according to embodiments of thepresent invention. There are a large number of conceivable gestures thatthe user can use to control actions of the device, and a non-exhaustivenumber of such gestures have been described above. At step S520, thegesture data, i.e. the user input, is translated into an input signal.At step S530, it is checked whether the input signal corresponds to apredefined or predetermined relative gesture. If not, the procedurereturns back to step S500. On the other hand, if yes (i.e. the inputsignal corresponds to a predefined gesture), a gesture based controlcommand is generated at step S570. At step S540, the user looks at ascreen or an information presentation area and at step S550 the user'sgaze is detected at the information presentation area. As mentionedabove, the step S540 is not a part of the method according toembodiments of the present invention. In step S560, a gaze point areaincluding a user's point of gaze on the screen or informationpresentation area. At step S580, an action corresponding to the relativegesture at the user's point of gaze is performed based on the gesturebased control command and the determined gaze point at the informationpresentation area.

With reference to FIG. 18, the device is waiting for user input in stepS500. In step S592, the user generates input by touching touchpad orpredefined area of touch-screen. The step S592 is not a part of themethod according to embodiments of the present invention. There are alarge number of conceivable gestures that the user can use to controlactions of the device, and a non-exhaustive number of such gestures havebeen described above. At step S520, the gesture data, i.e. the userinput, is translated into an input signal. At step S530, it is checkedwhether the input signal corresponds to a predefined or predeterminedrelative gesture. If not, the procedure returns back to step S500. Onthe other hand, if yes (i.e. the input signal corresponds to apredefined gesture), a gesture based control command is generated atstep S570. At step S540, the user looks at a screen or an informationpresentation area and at step S550 the user's gaze is detected at theinformation presentation area. The step S540 is not a part of the methodaccording to embodiments of the present invention. In step S560, a gazepoint area including a user's point of gaze on the screen or informationpresentation area is determined. At step S580, an action correspondingto the relative gesture at the user's point of gaze is performed basedon the gesture based control command and the determined gaze point atthe information presentation area.

With reference to FIG. 19, the device is waiting for user input in stepS500. In step S594, the user generates input by making a gesture withone or more of his or hers fingers and/or at least one hand. The stepS594 is not a part of the method according to embodiments of the presentinvention. There are a large number of conceivable gestures that theuser can use to control actions of the device, and a non-exhaustivenumber of such gestures have been described above. At step S520, thegesture data, i.e. the user input, is translated into an input signal.At step S530, it is checked whether the input signal corresponds to apredefined or predetermined relative gesture. If not, the procedurereturns back to step S500. On the other hand, if yes (i.e. the inputsignal corresponds to a predefined gesture), a gesture based controlcommand is generated at step S570. At step S540, the user looks at ascreen or an information presentation area and at step S550 the user'sgaze is detected at the information presentation area. The step S540 isnot a part of the method according to embodiments of the presentinvention. In step S560, a gaze point area including a user's point ofgaze on the screen or information presentation area is determined. Atstep S580, an action corresponding to the relative gesture at the user'spoint of gaze is performed based on the gesture based control commandand the determined gaze point at the information presentation area.

With reference to FIG. 21, a further implementation of the presentinvention will be discussed. A gaze tracking module (not shown) and auser input means 900 are implemented in a vehicle (not shown). Theinformation presentation area (not shown) may be a heads-up display oran infotainment screen. The input means 900 may be one or two separatetouch pads on the backside (for use with the index finger/s) or on thefront side (for use with the thumb/s) of the steering wheel 910 of thevehicle. A control module 950 is arranged in a processing unitconfigured to be inserted into a vehicle or a central processing unit ofthe vehicle. Preferably, the control module is a control module asdescribed with reference to FIGS. 13a -13 c.

With reference to FIG. 22, another implementation of the presentinvention will be discussed. A gaze tracking module (not shown) and aninformation presentation area (not shown) are implemented in a wearablehead mounted display 1000 that may be designed to look as a pair ofglasses. One such solution is described in U.S. Pat. No. 8,235,529. Theuser input means 1010 may include a gyro and be adapted to be worn bythe user 1020 on a wrist, hand or at least one finger. For example, theinput means 1010 may be a ring with a wireless connection to the glassesand a gyro that detects small movements of the finger where the ring isworn. The detected movements representing gesture data may thenwirelessly be communicated to the glasses where gaze is detected andgesture based control commands based on the gesture data from the inputmeans is used to identify and execute user action. Preferably, a controlmodule as described with reference to FIG. 13a-13c is used with thisimplementation.

With reference to FIG. 23, an implementation of the present inventionwill be discussed. In this implementation, the user 1120 is able tocontrol a computer device 1100 at least partly based on an eye-trackingsignal which describes the user's point of regard x, y on a informationpresentation area 1140 and based on user generated gestures, i.e. amovement of at least one body part of the user can be detected,generating gesture based control commands via user input means 1150. Inthis embodiment, the user 1120 can generate the gesture based controlcommands by performing gestures above or relative the keyboard of thecomputer device 1100. The input means 1140 detects the gestures, forexample, using an optical measurement technique or capacitivemeasurement technique. Preferably, a control module as described withreference to FIG. 13a-13c is used with this implementation and may bearranged in the computer device 1100. The computer device 1100 may, forexample, be any one from the group of a personal computer, computerworkstation, mainframe computer, or a handheld device such as a cellphone, portable music player (such as e.g. an iPod), laptop computers,computer games, electronic books and similar other devices. The presentinvention may also be implemented in an “intelligent environment” where,for example, objects presented on multiple displays can be selected andactivated. In order to produce the gaze tracking signals, a gaze trackerunit (not shown) is included in the computer device 1100, or isassociated with the information presentation area 1140. A suitable gazetracker is described in the U.S. Pat. No. 7,572,008, titled “Method andInstallation for detecting and following an eye and the gaze directionthereof”, by the same applicant, which hereby is incorporated in itsentirety. While this specification contains a number of specificembodiments, these should not be construed as limitation to the scope ofthe present invention or of what may be claimed, but rather asdescriptions of features specific to exemplary implementations of thepresent invention. Certain features that are described in thisspecification in the context of separate implementations can also beimplemented in combinations in a single implementation. Conversely,various features that are described in the context of a singleimplementation can also be implemented in multiple implementationsseparately or in any suitable sub-combination. Moreover, althoughfeature may be described above as acting in certain combinations or eveninitially claimed as such, one or more features from a claimedcombination can in some cases be excised from the combination, and theclaimed combination may be directed to a sub-combination or variation ofa sub-combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as require such operations beperformed in the particular order shown or in sequential order, or thatall illustrated operation be performed to achieve desirable results. Incertain circumstances, multitasking and parallel processing may beadvantageous. Moreover, the separation of various system components inthe implementation described above should not be understood as requiringsuch separation in all implementations, and it should be understood thatthe described program components and systems can generally be integratedtogether in a single software product or packaged into multiple softwareproducts.

1. A system for user interaction with an information presentation areaof a vehicle, the system comprising: input means adapted to detectuser-generated gestures, the input means comprising at least onetouchpad arranged on a steering device of a vehicle or adapted to beintegrated in a steering device of the vehicle, wherein user-generatedgestures include touch inputs of the user; a gaze tracking moduleadapted to detect gaze data of a viewer of the information presentationarea; and a control module configured to: acquire user input from theinput means and gaze data signals from the gaze tracking module;determine at least one user-generated gesture-based control commandbased on the user input; determine a gaze point area on the informationpresentation area including the user's gaze point based on at least thegaze data signals; and execute at least one user action manipulating aview presented on the graphical information presentation area and atleast one user-generated gesture-based control command; wherein the useraction is executed with the determined gaze point area as a startingpoint; wherein the information presentation area comprises a heads-updisplay (HUD) or an infotainment display of the vehicle.
 2. The systemof claim 1, wherein the input means comprises two separate touchpads. 3.The system of claim 1, wherein the touchpad is situated on a front ofthe steering wheel or on a back of the steering wheel.
 4. The system ofclaim 1, wherein the control module is further configured to checkwhether the user-generated gestures provided to the input meanscorrespond to a predefined or predetermined relative gesture.
 5. Thesystem of claim 1, wherein the control module comprises a gesturestorage unit for storing a library or list of predefined gesturesagainst which the user-generated gestures are compared.
 6. The system ofclaim 5, wherein each predefined gesture corresponds to a specificinput.
 7. The system of claim 4, wherein the control module further usesgaze data to interpret the user-generated gestures.
 8. The system ofclaim 1, wherein the control module is configured to be inserted into avehicle or a central processing unit of the vehicle.
 9. The system ofclaim 1, wherein the input means further comprise at least one of: anIR-sensor, voice activated input means, a detection device of bodygestures, or proximity-based input.
 10. A vehicle comprising: aninformation presentation area comprising a heads-up display (HUD) or aninfotainment system; and a system for user interaction with theinformation presentation area; wherein the system comprises: input meansadapted to detect user-generated gestures, the input means comprising atleast one touchpad arranged on a steering device of the vehicle, whereinuser-generated gestures include touch inputs of the user; a gazetracking module adapted to detect gaze data of a viewer of theinformation presentation area; and a control module configured to:acquire user input from the input means and gaze data signals from thegaze tracking module; determine at least one user-generatedgesture-based control command based on the user input; determine a gazepoint area on the information presentation area including the user'sgaze point based on at least the gaze data signals; and execute at leastone user action manipulating a view presented on the graphicalinformation presentation area and at least one user-generatedgesture-based control command; wherein the user action is executed withthe determined gaze point area as a starting point.